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I . INTRODUCTION 


The  recent  draft  of  a revision  to  Department  of  Defense 
(DOD)  Directive  5000.1  suggests  that  the  Pentagon  is  moving 
into  a new  era  of  program  management.  The  groundwork  for 
this  revision  to  David  Packard's  earlier  attempt  to  establish 
a firm  policy  base  for  the  DOD  procurement  has  been  well 
laid  by  the  reports  of  the  Commission  on  Government  Procure- 
ment and  the  Acquisition*  Advisory  Group,  and  by  Office  of 
Management  and  Budget  Circular  A-109.  This  modification  of 
Directive  5000.1  places  greater  emphasis  on  the  necessity 
to  evaluate  proposed  new  weapon  systems  on  the  basis  of 
reconciled  mission  needs  and  requires  an  acquisition  process 
in  which  alternative  concepts/solutions  are  investigated  in 
detail.  The  degree  of  program  oversight  by  the  Office  of  the 
Secretary  of  Defense  (OSD)  as  proposed  has  been  increased  by 
the  addition  of  another  decision  milestone,  "DSARC  0”,  at 
which  point  the  initiation  of  the  conceptual  effort  will 
require  Secretary  of  Defense  approval. 

Faced  with  mounting  public  pressure  for  greater  control 
of  costs  and  performance  following  some  notable  examples  of 
forecasting  errors  like  the  C-5A  and  LHA  programs,  DOD 
appears  to  have  responded  by  striving  to  exorcise  every 
possible  bit  of  uncertainty  from  the  conduct  of  weapon 
system  acquisitions  prior  to  the  committment  of  major 
resources.  The  desired  result  would  be  the  procurement  of 
systems  which  satisfy  basic  operational  requirements  by  a 


firm  date  and  at  a reasonable  and  predetermined  price. 

Although  written  several  years  prior  to  the  proposed 

policy  update,  J.  Ronald  Fox’s  words  have  an  ' I 've-seen-it- 

all-before'  air  to  them  as  he  comments  in  Arming  America  on 

the  changing  patterns  of  military  acquisition  management. 

Since  1969  there  have  been  further  attempts  to 
improve  other  aspects  of  weapons  acquisition 
management.  These  include  the  establishment  of 
"milestones''  for  major  programs  (i.e.,  dates  for 
the  accomplishment  of  crucial  program  objectives) ; 
the  synchronization  of  management  personnel 
changes  with  program  milestones;  a reduction  in 
the  numbers  of  letters  of  contract;  the  improvement 
of  training  programs  for  program  management 
officials;  a formal  review  procedure  for  each  new 
weapon  system's  operational  specifications.  The 
object  of  the  review  would  be  to  insure  that 
program  requirements  are  reasonable  and  attainable, 
and  that  prototype  systems  are  developed  and 
tested  before  a production  program  is  approved. 

Serious  doubts  about  the  successful  implementation 
of  this  proposal  have  troubled  Defense  Department 
observers,  however.  . .Many  procurement  officials 
in  the  Defense  Department  are,  after  all,  aware 
that  the  present  system  has  faults.  But  despite 
a steady  succession  of  studies  and  recommendations, 
the  procurement  process  has  remained  impervious  to 
structural  reform. ^ 

Certainly  only' time  will  prove  Fox  right  or  wrong,  but  there 
is  some  evidence  submitted  by  OSD  that  cost  control  has  at 
least  received  greater  attention  among  the  major  programs. ^ 


1 Fox,  J.  R.,  Arming  America , p.  456,  Harvard  University 
Press,  1974 

2 in  a memorandum  originating  in  the  Office  of  the 
Director,  Planning  and  Evaluation  (OSD)  dated  9 August  1976, 
it  was  reported  that  weapon  system  cost  growth  had  declined 
generally  in  the  last  twelve  months. 
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The  purpose  of  this  study  was  twofold.  The  first  was 
to  establish  an  historical  perspective  of  major  programs 
reaching  fruition  in  the  early  to  mid-1970's  which  will 
provide  a means  of  comparison  for  future  studies  of  program 
success,  and  thus  perhaps  support  Fox's  view.  While  the 
meaning  of  "program  success"  might  well  vary,  it  is  here 
defined  as  the  degree  to  which  a program  met  initial 
estimates  of  cost,  schedule  and  performance  criteria.  The 
first  goal  was  met  essentially  by  listing  those  programs 
studied  with  their  associated  degrees  of  success  in 
Appendix  C. 

The  second  desired  result  was  to  investigate  the  validity 
of  certain  hypotheses  about  program  management.  Specific- 
ally, the  approach  taken  in  this  case  was  to  highlight  gross 
differences  in  program  outcomes  on  the  basis  of  shared 
program  characteristics , such  as  program  size,  initial 
funding  and  technical  advance.  Associated  with  this  aim  was 
an  extensive  correlation  analysis  ±n  which  the  predictability 
of  program  success  was  determined  from  program  initial 
conditions  when  certain  variables  such  as  service  sponsor, 
weapon  system  type  and  life-cycle  phase  were  controlled. 

A.  UNCERTAINTY  IN  WEAPONS  SYSTEM  DEVELOPMENT 

A.  W.  Marshall  and  W.  H.  Meckling  of  the  Rand  Corporation 
addressed  the  problem  of  the  predictability  of  cost, 
schedule  and  performance  estimates  associated  with  development 
programs  in  a report  prepared  for  the  U.  S.  Air  Force  in  1959. 


Their  conclusions  regarding  the  uncertainty  of  new  develop- 
ments were: 

1.  Early  estimates  of  important  parameters  are  usually 
quite  inaccurate  in  two  respects.  First  of  all,  they 
are  biased  toward  overoptimism,  and  secondly,  their 
error  is  subject  to  a large  variation. 

2.  The  accuracy  of  estimates  is  a function  of  the 
stage  of  the  system  life-cycle  at  which  time  the 
appraisal  was  made. 

The  authors  suggested  a relationship  between  developmental 
estimates  and  the  military  world  of  R&D  decision  making, 
but  little  if  any  analysis  was  provided  in  their  report 
other  than  to  say  that  a problem  in  the  way  estimates  are 
generated  does  exist.  Also,  the  number  of  programs 
reviewed  appeared  rather  small  (22)  and  they  were  all  Air 
Force  sponsored. 

Following  the  above  report  was  the  work  of  M.  J.  Peck 
and  F.  M.  Scherer,  The  Weapons  Acquisition  Process:  An 
Economic  Analysis  which  was  published  in  1962.  This  bell- 
wether study  of  the  business  of  arms  procurement  improved 
noticeably  on  Marshall  and  Meckling's  efforts  to  gain  an 
understanding  of  the  unpredictable  nature  of  weapons 
acquisition. 

A major  thesis  of  this  study  is  that  the  weapons 
acquisition  process  is  characterized  by  a unique 
set  of  uncertainties  which  differentiates  it  from 
other  economic  activity.  To  be  sure,  uncertainty 
is  a pervasive  feature  of  all  economic  activity, 


and  most  of  the  uncertainties  in  weapons  acquisition 
have  their  commercial  counterparts.  But  there  is  a 
uniqueness  in  both  the  magnitude  and  diverse  sources 
of  uncertainty  in  weapons  acquisition. 3 

The  operational  definition  of  uncertainty  was  provided  by 

the  authors  as  "the  relative  unpredicatability  of  the  outcome 

of  a contemplated  action". 

Herein  lies  a problem  which  has  confronted  those 

researchers  who  have  attempted  to  subject  the  success  of 

program  outcomes  to  statistical  analysis  based  upon  a 

comparison  of  initial  estimates  and  demonstrated  performance. 

Peck  and  Scherer  summarized  the  difficulty  in  this  way. 

Let  us  begin  by  attempting  to  ascertain  the  degree  to 
which  the  outcomes  of  weapons  programs  are  predictable. 

A rough  measure  is  obtained  by  examining  three  crucial 
kinds  of  weapons  program  outcome  predictions:  (1)  qua- 

lity, or  the  expected  technical  performance  and  the 
reliability  of  the  resulting  weapon  system;  (2)  develop- 
ment time,  or  the  interval  between  the  start  of  a 
development  effort  and  the  availability  of  operational 
weapons;  and  (3)  the  cost  of  development.  Characteris- 
tically, predictions  of  these  variables  are  made  by 
contractors  and/or  buying  agencies  early  in  a weapons 
program  and  are  recorded  in  written  documents.  The 
predictability  of  the  variables  can  be  determined  by 
comparing' the  early  predictions  with  actual  quality, 
time  and  cost  outcomes. 

Nevertheless,  comparisons  of  this  sort  are  properly 
viewed  with  considerable  reservation,  for  there  are 
extreme  difficulties  in  obtaining  satisfactory  data. 
Neither  the  original  estimates  nor  the  final  outcomes 
are  completely  comparable  between  programs,  nor  are 
they  even  necessarily  representative  of  the  true  state 
of  affairs  within  a given  program. ^ 


3 Peck,  M.  J.  and  Scherer,  F.  M.,  The  Weapons  Acquisition 
Process:  An  Economic  Analysis,  p.  17,  Research  Division, 
Graduate  School  of  Business  Administration,  Harvard  University, 
1962 

4 Ibid. , p.  19 
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The  reasons  given  did  overlap  to  an  extent  with  those 
reported  by  Marshall  and  Meckling. 

1.  Initial  predictions  were  made  at  varying  points 
in  the  maturity  of  different  programs. 

2.  Initial  estimates  were  optimistically  low  and 
in  some  cases  deliberately  inaccurate. 

3.  Performance  outcomes  could  not  be  measured 
quantitatively . 

4.  Time  of  availability  outcomes  were  qualitatively 
assessed. 

5.  Cost  outcomes  were  blurred  by  arcane  bookkeeping 
procedures,  whereby  developmental  costs  were  buried 

in  production  expenditures  or  post-production  support. 

In  a criticism  of  the  Selected  Acquisition  Report  (SAR) , the 

Blue  Ribbon  Defense  panel  questioned  the  validity  of  before- 

and-after  comparisons  futher. 

The  basic  approach  to  the  SAR  is  the  establishment  of 
a baseline  of  estimated  costs,  schedules  and  technical 
performance,  and  the  subsequent  measurement  of  the 
present  status  against  this  baseline.  . .Successfully 
predicting  the  course  of  development  of  a new  weapon 
system  is  uncertain  at  best.  The  long  period  of  time 
involved  introduces  unpredictable  changes  as  outside 
events  and  circumstances  shift  during  the  five  to  nine 
years  it  usually  takes  to  acquire  a new  weapon.  The 
development  process  itself  contains  hidden  unknowns. 

The  original  estimates  of  cost,  schedule  and  technical 
performance  of  a weapon  system  can  be  made  with  con- 
siderable skill  and  with  total  honesty,  but  they  remain 
only  estimates,  the  worth  of  which  can  be  determined 
only  by  the  future  unfolding  of  events.  The  SAR’s 
tend  to  treat  the  original  estimates  as  accurate  pre- 
dictions and  to  measure  subsequent  events  in  the  dev- 
elopment against  the  standard  of  the  original  estimates. 
There  are  two  serious  consequences  of  this  procedure. 


. 


...The  overwhelming  concentration  now  appears  to  be 
on  maintenance  of  the  costs  and  schedule  within  the 
original  estimate.  Concern  with  the  quality  of  the 
weapon  and  its  ability  to  perform  an  essential  mission 
are  not  presented  in  the  SAR.  . .Inhibition  against 
change  is  the  second  serious  consequence  of  the  present 
SAR  system. 5 

The  picture  is  not  so  bleak  as  one  would  be  led  to 
believe.  Peck  and  Scherer  pointed  out  the  other  side  of 
the  ledger  as  well  in  their  analysis  of  the  processes  of 
weapons  development. 

To  examine  the  effectiveness  of  weapons  program  execution 
in  an  operational  way  requires  a meaningful  criterion  of 
effectiveness.  One  possibility  is  to  evaluate  actual 
time,  cost  and  quality  results  against  original  predic- 
tions. In  particular,  original  time,  cost  and  quality 
predictions  do  not  necessarily  reflect  optimal  choices. 
Still,  the  analysis  of  variances  from  original  predic- 
tions has  several  values.  For  one,  it  is  useful  method 
of  pinpointing  the  problems  which  affect  program  out- 
comes. Furthermore,  to  the  extent  that  major  national 
security  decisions  are  made  on  the  basis  of  original 
predictions,  sizeable  variances  from  those  predictions 
may  imply  a serious  misallocation  of  resources. 

A greater  implication  of  the  position  that  comparative  studies 
are  of  no  utility  is  that  without  comparisons,  the  effective- 
ness of  reforming  activities  within  the  weapon  system 
acquisition  process  must  be  left  entirely  to  intuition. 

Peck  and  Scherer  did  attempt  to  pinpoint  problems 
affecting  program  outcomes  by  correlating  Marshall  and  Meck- 
ling's  criteria  of  program  success  with  a largely  qualitative 


5 Blue  Ribbon  Defense  Panel,  Report  to  the  President  and 
the  Secretary  of  Defense  on  the  Department  of  Defense,  Vol.  1, 


p.  125,  U.S.  Government  Printing  Office,  1970 

6 Peck,  M.  J.  and  Sherer,  F.  M. , The  Weapons  Acquisition 
Process:  An  Economic  Analysis,  p.  428,  Research  Division, 
Graduate  School  of  Business  Administration,  Harvard  University, 
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collection  of  explanatory  variables.  The  results  were 
characterized  by  a lack  of  statistical  significance  which 
might  be  explained  by  a reliance  on  parametric  procedures. 
However,  the  results  did  indicate  that  definite  relation- 
ships existed  between  program  characteristics  externally 
imposed  on  the  project,  such  as  urgency,  the  importance  of 
minimizing  costs,  state-of-the-art  exploitation  and  program 
results. 7 

This  study  was  conceived  in  part  as  a redirected  con- 
tinuation of  the  above  aspect  of  Peck  and  Scherer's  work; 
redirected  in  the  sense  that  the  choice  of  independent 
variables  was  limited  to  quantitative  measures  of  a program 
(e.g. , unit  cost,  total  program  cost,  development  expendi- 
tures and  program  length)  which  were  then  applied  to  a 
larger  sample  of  programs  using  nonparametric  analyses. 

However,  more  importantly  this  thesis  has  attempted  to 
ascertain  the  validity  of  certain  general  beliefs  of  the 
nature  of  program  management  and  its  effect  on  program  out- 
comes. Four  separate  hypotheses  have  been  presented,  each 
in  its  own  right  questioning  the  predictability  of  project 
success  when  considered  as  the  result  of  a single  determinant 
variable.®  In  short,  the  thesis  has  erected  a model  of  the 
acquisition  process  in  which  the  management  skills  and 

7 Peck,  M.  J.  and  Sherer,  F.  Mt,  The  Weapons  Acquisition 
Process:  An  Economic  Analysis,  ch.  , Research  Division, 
Graduate  School  of  Business  Administration,  Harvard  University, 
1962 

® A fifth  hypothesis  was  posed  to  test  the  data  source 
for  bias. 


practices  of  a system's  program  office  staff  or  individual 
manager  have  been  ignored  as  predictors  of  program  results. 

The  model  was  further  refined  by  suppressing  the  externally 
imposed  noise  of  program  contractions  (expansions)  and 
economic  escalation.  The  hypothesis  testing  served  to 
illustrate  one  dimensional  differences  in  acquisition  programs 
(i.e.,  cost  growth  differences  in  programs  differentiated 
by  project  cost  alone) . 

In  addition,  a series  of  correlation  tests  were  conducted 
in  which  the  fundamental  differences  in  program  outcomes  were 
further  tested  by  controlling  additional  program  character- 
istics such  as  service  sponsor  or  weapon  system  type.  The 
purpose  of  the  correlation  was  twofold,  first  to  establish 
whether  the  controlling  process  accented  the  gross  differences 
identified  by  the  hypothesis  testing,  and  secondly,  to  identify 
among  the  program  facets  of  size,  length  and  risk,  effective 
predictors  of  program  outcomes. 


II . METHODOLOGY 


A.  SELECTED  ACQUISITION  REPORT 

Recognizing  the  problem  of  how  program  maturity  may 
distort  the  accuracy  of  early  forecasts,  it  was  considered 
essential  that  some  instrument  adjust  initial  estimates  for 
all  programs  to  a common  beginning,  and  also  to  allow  for  the 
adjustment  of  these  forecasts  in  the  light  of  a lengthy 
program  life.  Such  an  instrument  was  the  Selected  Acquisition 
Report  (SAR) . 

The  SAR  serves  as  a comprehensive,  summary  acquisition 
status  report  for  the  "highest  levels  of  management  within 
the  Department  of  Defense  and  (is)  the  basis  for  reports  to 
the  Congress  and  other  government  agencies  including  the 
Government  Accounting  Office  (GAO)".9  Compiled  at  the  close 
of  each  quarter,  the  SAR  tracks  the  success  of  each  major 
weapons  system  acquisition  in  the  areas  of  cost,  technical 
performance  and  schedule. 

In  each  of  these  areas,  the  report  offers  a comparison 
between  current  estimates  for  program  success  and  the  develop- 
ment, or  initial,  estimates.  Sandwiched  between  these  two 
expectations  is  yet  a third,  the  approved  program  estimate 
which  in  practive  varies  only  slightly,  if  at  all,  from  the 
development  estimate.  The  development  estimate  is  established 


9 Secretary  of  the  Navy  Instruction  7000.3 


i 


F.  • 


i 
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\ 
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once  the  Secretary  of  Defense  grants  his  approval  for  the 
program  to  move  into  full  scale  development,  and  generally 
has  as  its  basis  the  Decision  Coordinating  Paper  (DCP) . 

The  development  estimate  for  procurement  may  be  reviewed  once 
the  production  contract  has  been  let,  but  otherwise,  the 
estimate  may  not  be  changed  without  permission  of  the  Assis- 
tant Secretary  of  Defense  (Comptroller) . Fox  points  out 
the  necessity  of  this  last  requirement  as  he  quotes  a DOD 
review  of  management  control  practices. 

This  problem  has  been  common  enough  to  acquire  the 
esoteric  but  descriptive  label,  "the  rubber  baseline". 
Translated,  this  means  that  the  contractor  managers 
would  periodically  make  retroactive  adjustments  to 
cost  estimates  and  schedules  so  that  plans  would 
equal  actuals.  . .Obscuring  overruns  in  this  fashion 
contributed  to  our  being  caught  short  on  funding  for 
two  major  programs  last  fiscal  year.10 

The  approved  program  estimate  is  also  based  upon  the  DCP,  or 

in  its  absence,  a Program  Memorandum  or  Program  Change 

Decision.  As  these  formal  statements  of  changes  in  program 

goals  are  forthcoming,  the  estimate  will  necessarily  be 

revised.  The  current  estimate  represents  the  program 

manager's  most  realistic  assessment  of  program  status.  As 

the  program  matures,  this  appraisal  will  more  and  more  equate 

to  demonstrated  performance  or  the  actual  completion  of 

I 

schedule  milestones.  The  31  March  1976  SAP  for  the  A-7E 
project  is  provided  as  an  example  in  Appendix  A. 

The  30  June  1976  SAR  was  the  principal,  primary  reference 

10  Fox,  J.  R. , Arming  America,  p.  413,  Harvard  University 
Press,  1974 
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source  of  project  data  for  this  thesis.  In  fact,  because  of 
the  completeness  of  the  report  and  the  clarity  of  its  format, 
whether  or  not  an  acquisition  program  was  included  under  the 


reporting  system  determined  whether  that  program  would  be 
included  in  the  analysis.  Thus,  this  determination  excluded 
a number  of  weapon  systems  presently  in  either  development  or 
procurement,  principally  in  the  areas  of  electronic  counter- 
measures and  naval  gunnery. 


B.  ANALYTIC  METHODS 


1 . Hypothesis  Testing 

1.  Large  programs  have  a tendency  to  incur  greater 
cost  and  schedule  growth  in  absolute  and  relative  terms  than 
do  smaller  programs. 

One  of  the  initial,  underlying  assumptions  of  this  thesis  was 
that  not  only  will  weapon  system  acquisitions  experience  cost 
growth,  but  that  for  system  with  large  program  costs,  the 
growth  will  be  correspondingly  greater  than  for  those  less 
expensive  projects.  Reasons  for  the  explanation  of  the  cost 
growth  phenomenon  have  been  postulated  again  and  again  by 
critics  of  military  procurement  (e.g,.  Peck  and  Sherer,  Fox). 
Not  the  least  frequently  observed  reason  is  that  of  over- 
optimism. It  would  seem  reasonable  that  large,  expensive 
programs  would  require  a certain  amount  of  ''selling"  on  the 
part  of  sponsors  to  ease  the  difficulties  of  higher  level 
DOD  and  Congressional  approval.  This  "selling"  often  took 
the  form  of  understating  the  eventual  program  cost,  and  with 


. 

' 
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it  the  technological  risk  involved  in  the  development,  not 
to  mention  the  date  on  which  the  system  enter  service. 
Whether  overoptimism  was  the  result  of  deliberate  policies 
on  the  part  of  the  sponsoring  agency  is  something  of  a moot 
point.  Peck  and  Scherer,  however,  did  point  out  that  the 
practice  has  been  accepted  as  a matter  of  fact  within  the 
defense  industry. 

Quite  commonly,  this  strategy  includes  submitting 
proposals  which  in  one  or  more  aspects  are  highly 
optimistic.  Indeed,  the  propensity  to  'but  into' 
attractive  new  programs  with  optimistic  quality, 
time,  and/or  cost  estimates  is  perhaps  as  much  as 
an  industry  practice  in  advanced  weapons  acquisition 
as  list  price  cutting  is  in  automobile  retailing,  or 
as  the  advertising  of  loss  leaders  is  in  department 
store  operations. 1! 

Concurrent  with  the  overoptimism  that  acompanies  large 
programs,  it  was  presumed  that  there  would  be  difficulties 
encountered  in  the  area  of  cost  control  simple  because  of 
program  size,  an  extension  of  the  belief  that  with  size  one 
inherits  waste  and  a certain  amount  of  uncontrollability. 
Unfortunately,  it  is  impossible  to  distinguish  explicitly 
between  optimism  or  waste  based  on  data  gathered  from  the 
SAR' s. 


2 . Programs  with  a larger  development  phase  budget 
have  a greater  tendency  to  achieve  initial  cost, 
performance  and  schedule  estimates. 

An  acedemic  approach  to  the  study  of  project  management  has 
stressed  the  importance  of  "front-end  loading"  the  acquisition 


H Peck,  M.  J.  and  Scherer,  F.  M.,  The  Weapons  Acquisition 
Process:  An  Economic  Analysis,  p.  413,  Research  Division, 
Graduate  School  of  Business  Administration,  Harvard  University, 
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programs . 

For  the  purpose  of  this  analysis,  those  programs  which  had 
more  than  one  prime  contractor  regardless  of  whether  the 
systems  were  in  procurement  or  development  were  considered 
multiple  source  programs.  In  the  case  of  air-craft  systems, 
if  there  were  competing  contractors  for  either  the  airframe 
or  engine  designs,  the  program  was  considered  to  have  a 
"second  source".  Frederick  Sherer  in  his  study  of  the 
economic  incentive  in  the  weapons  system  acquisition  process 


The  effect  was  originally  suggested  to  this  student 
by  Professor  Melvin  Kline,  U.  S.  Naval  Postgraduate  School. 
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documented  the  significant  cost  reductions  which  developed 
during  the  joint  production  of  military  aircraft  in  World 
War  II.  Although  not  in  as  much  detail  as  Scherer,  Fox 
touched  upon  the  benefits  of  "second  sourcing",  reporting 
that  "it  would  be  reasonable  to  assume  a 25%  reduction  in 
contract  price  can  be  achieved  in  many,  if  not  most,  cases 
when  competition  is  introduced. "13  it  would  appear  that 
this  is  one  area  in  which  weapons  procurement  responds 
to  traditional  market  forces.  The  fourth  hypothesis  seeks 
to  determine  first,  whether  the  evident  cost  successes  do 
exist  in  a significant  number  of  cases,  and  second,  whether 
the  advantage  of  multiple-source  contracting  is  extended 
to  the  areas  of  performance  and  schedule. 

5.  The  prospect  of  program  success  dimenishes  with 
increasing  maturity  among  programs. 

This  last  hypothesis  was  posed  in  order  to  test  the  validity 
of  the  SAR  in  overcoming  the  criticisms  alluded  to  earlier 
about  the  non'comparability  of  programs  in  differing  stages 
of  their  life-cycle.  For  example,  early  in  the  life  of  a 
program,  before  the  realities  of  the  management  or  technical 
task  at  hand  are  more  apparent,  one  would  expect  that  current 
estimates  of  project  outcomes  would  agree  with  initial 
assessments.  But,  as  the  program  achieves  greater  maturity, 
and  uncertainties  arise,  the  gap  between  the  two  expectations 

Fox,  J.  R.,  Arming  America,  p.  256,  Harvard  University 
Press,  1974 
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would  be  sure  to  widen. 

Each  hypothesis  was  tested  using  a one-tailed  Mann- 
Whitney  U Test  (a  nonpar ame trie  one  way  analyses  of  variance) 
in  which  the  means  of  program  outcomes  were  compared.  For 
the  most  part,  the  sample  of  48  programs  would  be  divided 
along  the  median  of  the  program  variable  in  question,  except 
in  the  case  of  the  fifth  hypothesis  where  whether  a program 
was  in  development  or  procurement  became  the  classifying 
criterion.  The  results  of  these  tests  are  presented  in 
Chapter  III. 

2 . Correlation 

As  stated  earlier,  associated  with  the  hypothesis  testing 
was  a desire  to  examine  the  correlation  between  the  output 
variables  of  cost,  performance  and  schedule,  and  initial 
program  estimates  of  program  length,  cost  and  technical  risk 
when  specific  program  characteristics  were  controlled  for 
explicitly.  The  thesis  employed  a nonparametric  correlation 
proceudre  offered  as  a subprogram  in  SPSS:  Statistical 
Package  for  the  Social  Sciences  by  Norman  H.  Nie,  et  al. 

The  following  characteristics  were  considered  and  controlled 
for  in  deriving  correlations  between  program  outcomes  and 
initial  estimates: 

1.  All  forty-eight  programs  together  without  any 
differentiation; 

2.  Programs  differentiated  by  the  maturity  of  their 
design  (i.e.,  whether  the  program  was  in  development  or 
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had  advanced  to  procurement) ; 

4.  Programs  differentiated  by  system  type,  ship, 
aircraft,  missile  or  miscellaneous  (tracked  vehicles, 
fire-control  systems,  etc.); 

5.  Programs  differentiated  by  service  sponsor.  Navy, 
Army,  or  Air  Force. 

The  first  category  of  'undifferentiated  programs'  was 
included  as  a control  feature  to  serve  as  a means  of  com- 
parison for  the  following  four  program  characteristics. 

Items  two  and  three  were  included  as  part  of  follow-up  tests 
to  the  verification  of  the  fourth  and  fifth  hypotheses. 

The  last  two  program  characteristics,  system  type  and  service 
sponsor,  were  included  because  they  are  frequently  used  as 
means  of  ordering  weapons  systems.  Also,  there  are  sub- 
stantial inter-service  differences  in  system  acquisition, 
and  as  a result,  there  are  natural,  virtually  instinctive 
inclinations  to  make  comparisons  between  the  programs  of 
each  service. 

; Additionally,  the  intercorrelations  among  program  out- 

comes were  examined  to  see  if  any  light  could  be  shed  on  the 
trade-offs  between  cost,  performance  and  schedule.  The 
correlation  results  are  presented  in  Chapter  III. 

C.  SELECTION  OF  INDEPENDENT  VARIABLES 
1 . Program  Cost 

For  the  purpose  of  the  analysis,  it  was  considered 
essential  that  each  of  the  forty-eight  programs  be  considered 

hi 


on  an  equal  footing,  that  is,  considered  from  the  same  base 
year.  In  this  way,  comparability  between  programs  on  the 
basis  of  total  program  cost  would  be  maintained.  The  base 
year  selected  was  1974  since  all  but  two  programs  (added  late 
in  the  thesis)  predated  that  year.  Each  of  the  programs 
reported  development  estimate  program  costs  in  their  own 
base  year  dollars,  which  in  turn  were  converted  to  1974 
dollars.  The  method  of  accomplishing  this  conversion  was 
not  very  satisfactory,  but  had  to  suffice  for  lack  of  a 
superior  alternative.  From  a Direct  Budget  Plan  (TOA) 

History  for  the  Department  of  the  Navy,  an  implicit  price 
deflator  for  the  fiscal  year  1964  through  1977  was  derived. 
The  unsatisfactory  aspects  of  this  source  are  evident  when 
one  considers  that  the  TOA  history  included  nonprocurement 
appropriations  such  as  OM&N  as  well  as  the  procurement 
related  appropriations,  RDT&E , SON,  etc.  The  U.  S.  Bureau 
of  Economic  Analysis  is  in  the  process  of  determining 
deflators  for  each  of  the  basic  defense  industries,  but 
results  will  not  be  forthcoming  until  late  1977. 

A copy  of  the  TOA  history  a:td  an  example  of  the  manner 
in  which  the  implicit  price  deflators  for  each  of  the 
program  base  years  was  derived  is  provided  in  Appendix  B. 

2 . Program  Unit  Cost 

Unit  cost  was  selected  as  an  independent  variable 
on  the  assumption  that  program  success  (primarily  in  the 
area  of  cost)  could  not  be  adequately  explained  by  program 


4 


24 


size  "alone.  For  example,  a program  line  the  A-7E  in  which 
the  Navy  is  procurring  a proven  aircraft  design  with  up- 
dated avionics,  and  the  F-14A  program  which  represents  a 
completely  new  weapon  system,  could  have  conceivably  identical 
program  costs  provided  sufficient  quantities  of  the  A-7E 
were  purchased.  However,  the  true  nature  of  both  programs 
would  be  obscured.  Of  the  two,  clearly  the  F-14A  with  a 
program  unit  cost  of  $16. 8M  (1974$)  is  the  more  ambitious 
undertaking,  and  the  lesser  unit  cost  of  the  A-7E,  $4.1M 
in  1974  dollars,  would  reflect  this.  As  in  the  case  of 
program  cost,  development  estimates  of  unit  costs  were 
translated  into  1974  dollars. 

3 . Development  Investment 

The  development  investment  for  each  program  was 
derived  as  the  ratio  of  the  level  of  development  spending 
to  total  program  as  foreseen  at  DSARC  II.  No  judgement  of 
the  manner  of  (or  purpose  for)  development  spending  was 
attempted.  tfhe  variable  was  intended  to  serve  as  a com- 
plement to  program  unit  cost,  that  is,  a measure  of  the 
technological  advance  of  the  project.  Some  difficulties 
were  encountered  in  those  reports  for  nuclear  powered 
men-of-war  where  propulsion  system  development  costs 
attributed  to  both  the  Navy  and  the  Energy  Research  Devel- 
opment Agency  (ERDA)  were  not  included  in  the  cost  section 
of  the  SAR's,  but  rather  in  a SAR  addendum.  Consequently, 
at  first  glance  a project  as  large  and  complex  as  NIMITZ 
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(CVAN  68)  appears  to  be  an  off-the-shelf  product,  despite 
the  fact  that  her  two  reactor  propulsion  plant  represents 
a first  for  the  surface  Navy  and  is  hardly  a trivial  tech- 
nological step.  Adjustments  were  made  for  cases  such  as 
this  by  adding  the  additional  estimated  development  cost 
to  both  the  level  of  development  funding  and  the  total 
program  cost. 

4 . Program  Length 

Of  all  the  explanatory  variables,  time  was  the  most 

subjectively  determined.  Initial  program  length,  measured 

ostensibly  from  the  initiation  of  the  project  to  an  initial 

operational  capability  date  (IOC)  proved  to  be  a bit 

troublesome  to  extract  from  the  SAR ' s . Earlier  researchers 

have  encountered  similar  problems. 

Time  of  availability  predictions  are  often  made  in 
terms  of  an  r initial  operational  capability'  - a 
decidedly  qualitative  concept.  Does  one  missile 
on  a lonely  launch  pad  constitute  an  'initial  opera- 
tional capability'?  Or  do  30  missiles  when  their 
reliability  in  time  of  need  is  doubtful?^4 

Fortunately,  the  term  IOC  is  a common  one  within  DOD  and  in 

a large  number  of  cases,  those  dates  were  reported  explicitly. 

In  the  case  of  Navy  shipbuilding  programs,  a simplifying 

rule  was  established  whereby  the  date  on  which  the  lead  ship 

of  the  entire  class  in  question  became  available  for 


14  Peck,  M.  J.  and  Scherer,  F.  M.,  The  Weapons  Acquisition 
Process:  An  Economic  Analysis,  p.  20,  Research  Division, 
Graduate  School  of  Business  Administration,  Harvard  University 


operational  deployment  was  chosen  arbitrarily  as  the  "IOC" 
date.  There  are  obvious  draw-backs  to  this  method  when  the 
entire  class  is  considered  as  any  student  of  ship  construc- 
tion can  point  out.  Namely,  the  greatest  schedule  slippages 
often  occur  during  follow-ship  production.  Still,  the 
availability  of  the  lead  ship  does  represent  the  realization 
of  a design,  and  in  order  to  maintain  some  parity  with  other 
systems  produced  more  rapidly,  the  consistency  of  the  above 
method  was  considered  essential. 

The  other  end-point  of  program  length,  the  initiation 
of  the  project,  was  even  more  difficult  to  determine.  In 
most  cases,  a date  corresponding  to  the  advent  of  the  system 
life-cycle  was  selected  as  the  starting  point.  More  often 
than  not,  this  date  was  the  date  of  approval  for  the  system 
requirements/characteristics.  There  was  a distinct  lack  of 
uniformity  noted  among  the  various  programs  in  the  reporting 
of  this  schedule  milestone.  In  a few  cases  programs  were 
considered  underway  with  an  all  inclusive  'commenced 
engineering  development'  or  'initiated  validation  phase'. 

D.  SELECTION  OF  DEPENDENT  VARIABLES 
1 . Cost  Growth  Factor 

Fortunately,  a recent  revision  of  the  SAR  system 
directed  that  development  and  current  cost  estimates  be 
reported  in  constant  year  dollars  as  well  as  in  escalated 
figures.  This  change  has  made  the  pinpointing  of  true  cost 


growth  that  much  easier.  It  came  as  something  of  a shock 
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to  discover  that  little  or  no  provision  for  escalation  was 
included  in  the  development  estimate  of  many  programs. 

Where  some  accounting  for  escalation  was  made,  invariably 
the  inflation  rate  was  unrealistically  low. 

The  office  of  the  Assistant  Secretary  of  Defense 
(Comptroller)  publishes  a "SAR  Program  Acquisition  Cost 
Summary"  (see  Appendix  C)  once  each  quarter’s  collection  of 
individual  reports  is  complete.  From  this  summary  it  is 
possible  to  determine  each  program’s  cost  growth  both  in 
dollars  and  as  a percentage  of  original  estimates.  In  each 
instance,  the  resultant  figures  are  adjusted  by  OSD  analysts 
to  negate  the  effects  of  inflation  and  program  quantity 
changes.  Unfortunately,  the  relative  measure  of  program 
cost  growth  is  related  def initionally  to  the  original 
program  cost  estimate  as  shown  below. 

Cost  growth  (%)  = Cost  Changes 

Development  Estimate  + Quantity  Changes 

(All  entries  are  in  base 
year  dollars) 

One  would  expect  a negative  correlation  between  cost  growth 
(%)  and  program  size,  expressed  above  as  the  'development 
estimate' . The  extent  of  the  correlation  would  depend  on  the 
degree  to  which  the  Cost  and  Quantity  Changes  varied  with 
total  program  cost.  To  aid  in  clarifying  the  effect  of 
program  size  on  cost  success,  it  would  be  helpful  to  have  the 
assistance  of  an  absolute  measure  of  growth,  namely  Cost 
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Growth  ($)  which  is  the  dollar  value  of  the  overrun.  This 
measure  is  determined  as  shown  below. 

Cost  Growth  ($)  = Cost  Growth (%)  -1  x Development 

Estimate 

Hence,  a negative  "growth"  would  indicate  an  underrun. 

2 . Performance  Achievement  Factor 

Navy  and  Air  Force  concepts  of  system  effectiveness 
were  presented  in  the  1960's  as  a measure  of  how  well  a 
system  performed  its  intended  function.  This  figure  of 
merit  was  derived  originally  as  a companior  to  cost  effective- 
ness in  the  hope  that  together  the  two  would  serve  to  deter- 
mine whether  a system  undergoing  design  would  satisfy  most 
efficiently  those  operational  requirements  which  spawned  it 
in  the  first  place.  Despite  interservice  differences  in  the 
wording  of  what  was  meant  by  system  effectiveness,  the 
measure  (expressed  as  a probability)  was  generally  conceeded 
to  have  three  basic  components; 

1.  the  performance  capability  of  the  system; 

2.  the  operational  readiness  or  availability 
that  is,  its  ability  to  start  performance  of  a 
mission  when  called  upon  to  do  so; 

3.  the  continued  capability  of  the  system  to 
perform,  or  its  mission  reliability  or  dependa- 
bility. 

In  time  however,  it  became  apparent  that  models  of  system 
effectiveness  were  not  as  useful  as  originally  conceived. 
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The  computations  were  messy  and  highly  subjective.  For 
example,  how  much  does  the  failure  of  a single  component 
degrade  the  effectiveness  of  a system?  Or  even  more  basic, 
what  constitutes  a failure?  Since  the  derived  probabilities 
were  by  no  means  "hard  numbers",  they  were  mistrusted  by 
managers,  and  the  computational  aspects  of  system  effective- 
ness fell  into  disuse. 

Given  that  an  evaluation  of  system  performance  is  essential 
to  any  analysis  of  program  success,  and  in  the  absense  of  a 
readily  available  performance  factor  counterpart  to  cost 
growth,  a simplistic  method  of  marking  performance  using  the 
three  basic  elements  of  system  effectiveness  where  possible 
was  devised.  From  SAR  data  on  the  technical/operational 
characteristics  of  the  system  in  question,  it  was  possible 
to  contrast  current  estimates  of  system  performance  with 
development  estimates.  Generally  speaking,  the  performance 
achievement  factor  was  determined  by  assessing  a weighted 
average  of  major  performance  areas  such  as  speed,  time-on- 
station,  maintainability,  system  accuracy,  etc.  A value  of 
performance  less  than  1.0  indicates  that  the  system  achieved 
better  results  than  originally  estimated.  An  example  of  how 
this  computation  was  made  is  provided  in  Appendix  C. 

3 . Schedule  Overrun  Factor 

As  in  the  case  of  attaining  performance  goals,  the 
schedule  overrun  factor  was  determined  by  the  ratio  of 
current  to  development  estimates  of  program  length.  However, 


III.  RESULTS 


A.  HISTORICAL  PERSPECTIVE 

Appendix  C has  listed  those  programs  from  which  program 
data  was  obtained  for  the  analysis.  With  the  exception  of 
POSEIDON,  all  the  programs  are  currently  included  under  the 
SAR  system.  (POSEIDON  reports  were  terminated  in  June  1975 
following  completion  of  system  deployment  and  the  subsequent 
disestablishment  of  the  program  office) . For  the  convenience 
of  the  reader,  the  programs  are  ordered  by  service  sponsor. 

An  examination  of  the  outcome  means  for  each  service  indicates 
that  perhaps  Navy  programs  enjoyed  a greater  achieved  per- 
formance at  greater  cost  and  shcedule  growth  than  did  Army 
and  Air  Force  projects.  A sample  comparison  between  outcome 
means  revealed  that  the  NAVY  - USAF  differences  in  all  three 
categories  were  significant,  while  the  pertinent  ARMY  - NAVY 
disparity  was  limited  to  the  area  of  cost  alone. 

The  results  of  the  NAVY  - USAF  comparison  are  not  fully 
understood.  Sample  comparisons  of  Navy  programs  (less  ship 
systems)  and  the  Air  Force  projects  shown  did  result  in  less 
significant  cost  differences.  Additionally,  when  missile 
systems  (which  historically  have  proven  to  be  technically 
more  successful  than  other  system  types)  were  excluded  from 
both  samples,  the  performance  difference  between  services 
was  still  evident.  No  explanation  for  this  is  readily 
apparent,  nor  was  there  any  system  type  peculiar  to  a 
single  service  that  could  account  for  the  Air  Force  advantage 


in  meeting  deployment  schedules 
B.  HYPOTHESIS  TESTING 

1.  Large  programs  have  a tendency  to  incur  greater  cost 
and  schedule  growth  than  do  smaller  programs. 

Table  3-1  displays  the  difference  in  the  means  of 
program  outcomes  between  large  and  small  projects.  The  column 
marked  ’significance’  indicates  the  degree  to  which  the 
differences  were  statistically  significant.  In  instances  of 
no  significance,  the  column  entry  is  NS.  Subsequent  tables 
in  this  section  are  arranged  similarly, 

TABLE  3-1 

COMPARISON  OF  PROGRAMS  BASED  ON  PROGRAM  SIZE 
LARGER  PROGRAMS  SIGNIFICANCE  SMALLER  PROGRAMS 


Cost  Growth (%) 

1.13 

0.003 

Cost  Growth (%) 

1.29 

Performance 

0.99 

NS 

Performance 

1.00 

Schedule  Slip 

. 1.16 

0.005 

Schedule  Slip 

1.44 

(%) 

(%) 

Schedule  Slip 

12.6 

0.005 

Schedule  Slip 

25.2 

(Mos. ) 

(Mos . ) 

Cost  Growth ($) 534 . OM 

0.006 

Cost  Growth ($) 163 . 7M 

The  results  appear  to  provide  mixed  support  for  the  first 
hypothesis.  Ignoring  the  differences  in  relative  cost  growth 
for  reasons  given  in  Chapter  II,  note  the  differences  in 
schedule  overrun.  Historically,  larger  programs  have  done 
significantly  better  in  meeting  original  schedules,  both  in 
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a relative  and  absolute  sense.  The  lack  of  significance 


observed  in  the  area  of  achieved  performance  is  a result 
that  will  be  repeated  elsewhere  in  this  chapter.  Overall, 
the  nonparametric  analyses  of  variance  were  ineffectual  in 
accounting  for  the  variance  in  system  performance. 

In  the  area  of  cost  growth ($),  larger  programs  did  not 
appear  to  hold  the  line  against  overruns.  The  reasons  for 
this  may  have  included  a tendency  for  the  managers  of  large 
(and  therefore  important)  programs  to  throw  money  at 
problems,  particularly  schedule  problems,  until  a solution 
is  found.  Once  a major  program  has  been  funded,  rarely 
has  it  had  to  do  without  additional  resources  if  required. 

2.  Programs  with  more  development  funding  have  a greater 
tendency  to  achieve  initial  cost,  performance  and 
schedule  estimates 


TABLE  3-2 

COMPARISON  OF  PROGRAMS  ON  BASIS  OF  LEVEL 
OF  DEVELOPMENT  FUNDING 


HIGHER  PROGRAMS 

SIGNIFICANCE 

LOWER  PROGRAMS 

Cost  Growth (%) 

1.15 

0.040 

Cost  Growth (%) 

1.27 

Performance 

0.99 

NS 

Performance 

1.00 

Schedule  Slip 
(%) 

1.24 

0.002 

Schedule  Slip 
(%) 

1.42 

Schedule  Slip 
(Mos. ) 

15.94 

NS 

Schedule  Slip 
(Mos . ) 

21.7 

Cost  Growth 

459. 9M 

NS 

Cost  Growth 

234.7 

($)  ($) 


Perhaps  the  most  important  comparison  in  Table  3 - 2 is  that 
which  contrasts  relative  cost  growth.  During  the  analysis 
of  program  data,  a very  high  (0.750-1.000)  positive  corre- 
lation was  found  to  exist  between  program  cost  and  develop- 
ment funding  in  a significant  number  of  cases.  It  was  likely 
that  the  two  were  measuring  essentially  the  same  character- 
istic, program  size.  Thus,  one  would  expect  Tables  3-1  and 
3 - 2 to  bear  a resemblance  to  one  another.  In  terms  of 
relative  cost  growth  and  schedule  slippage,  this  similarity 
exists.  What  is  crucial  about  the  results  of  this  second 
sample  comparison  is  that  the  level  of  development  funding 
has  a significant  negative  impact  on  relative  cost  growth. 
Bearing  in  mind  that  the  development  cost  is  also  an  indirect 
measure  of  program  size,  this  is  the  first  instance  of  an 
indication,  not  confounded  by  definitions  of  the  variables, 
that  suggests  program  size  may  inhibit  the  degree  of  cost 
growth.  Further  signs  of  the  efficacy  of  early  and  sub- 
stantial investments  are  the  differences  in  relative  schedule 
slippage  and  the  absence  of  any  significance  to  the  difference 
between  the  absolute  cost  growth  for  each  sample. 

3 . Programs  with  perceived  technical  risks  tend  to  incur 
greater  cost  and  schedule  growth  than  do  less  risky 
programs 


TABLE  3-3 


COMPARISON  OF  PROGRAMS  ON 
BASIS  OF  DEVELOPMENT  INVESTMENT 


HIGHER  PROGRAMS 

SIGNIFICANCE 

LOWER  PROGRAMS 

Cost  Growth (%) 

1.18 

NS 

Cost  Growth (%) 

1. 

24 

Performance 

0.97 

NS 

Performance 

0. 

97 

Schedule  Slip 
(%) 

1.32 

NS 

Schedule  Slip 
(%) 

1. 

33 

Schedule  Slip 
(Mos.  ) 

19.4 

NS 

Schedule  Slip 
(Mos . ) 

18 

.2 

Cost  Growth ($) 

349. 8M 

NS 

Cost  Growth ($) 

340 

. 3M 

The  weakness  of  development  investment  as  a explanatory 
variable  is  apparent  in  Table  3-3.  No  significance  can 
be  attributed  to  the  difference  in  outcomes  between  programs 
with  a higher  development/committment  and  those  with  less. 

A second  comparison  between  the  extremes  of  each  group,  the 
highest  and  lowest  quartiles,  was  made  with  similar  null 
results . 

4.  Multiple-source  contracted  programs  enjoy  an  advan- 
tage in  program  success  over  sole-source  contracted 


programs 


r i 


TABLE  3-4 

COMPARISON  OF  PROGRAMS  ON  BASIS  OF 
PRODUCTION/DEVELOPMENT  SOURCES 


SOLE-SOURCE 

SIGNIFICANCE 

MULTIPLE  SOURCE 

Cost  Growth (%) 

1.21 

0.040 

Cost  Growth (%) 

1.14 

Performance 

0.99 

NS 

Performance 

1.02 

Schedule  Slip 
(%) 

1.30 

NS 

Schedule  Slip 
(%) 

1.44 

Schedule  Slip 
(Mos. ) 

17.8 

NS 

Schedule  Slip 
(Mos . ) 

24.7 

Cost  Growth ($) 

382. 3M 

0.049 

• Cost  Growth  ($) 

103. 

The  findings  of  Peck  and  Scherer  were  sustained  by  the  Mann- 
Whitney  test.  In  terms  of  relative  and  absolute  cost  growth, 
second-source  programs  did  better  than  their  sole-source 
counterparts.  Although  a small  class  of  projects  (7) , the 
multiple  source  programs  included  Army  and  Navy  projects  of 
all  system  types.  In  regard  to  the  achieved  performance  of 
both  program  types,  this  study  found  no  significance  in  the 
difference  of  program  outcomes.  This  is  at  variance  with  the 
findings  of  Rubin  and  Marquis'  who  determined  that  "sole  source 
contractors  do  significantly  better  technically  than  contrac- 
tors who  were  selected  by  some  form  of  competitive  procedure . 
Rubin  and  Marquis'  study  is  not  precisely  comparable  to  the 

16  Sloan,  Alfred  P.,  School  of  Management,  Massachussetts 
Institute  of  Technology,  Report  #192-66,  Critical  Decisions  in 
the  Initiation  of  Development  Projects,  Rubin,  I.  M.  and 
Marquis*,  D.  G.,  June  1966 


37 


one  undertaken  here  in  that  only  development  projects  in 
which  the  technical  performance  was  evaluated  subjectively 
by  management  and  engineering  personnel,  were  considered. 
However,  the  contention  that  greater  technical  performance 
can  be  achieved  in  the  absence  of  competition  was  not 
supported. 

5 . The  prospect  of  program  success  diminishes  with 
increasing  maturity  among  programs 

TABLE  3 - 5 (A) 

COMPARISON  OF  PROGRAMS  BASED 
ON  LIFE-CYCLE  DIFFERENCES 


PROGRAMS  IN  PROCUREMENT 

SIGNIFICANCE 

PROGRAMS  IN  DEVELOPMENT 

Cost  Growth (%) 

1.22 

NS 

Cost  Growth (%) 

1.18 

Performance 

0.98 

NS 

Performance 

1.00 

Schedule  Slip 

1.33 

NS 

Schedule  Slip 

1.29 

(%) 

(%) 

Schedule  Slip 

18.7 

NS 

Schedule  Slip 

17.5 

(Mos.)  • • 

(Mos . ) 

Cost  Growth ($) 

460. 6M 

0.012 

Cost  Growth ($) 

213. 9M 

Table  3-5  (a)  reveals  that  on  a gross  level  the  SAR  is  able  to 
minimize  the  differences  among  programs  that  arise  because 
of  life  cycle  disparities.  With  the  exception  of  program 
growth  in  dollars,  programs  in  procurement  and  those  still  in 
development  exhibited  similar  success.  The  exception  of 
program  growth  is  indicative  that  either  uncertainties  in 
schedule  or  producibility  do  still  exist,  and  that  the 
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development  phase  has  not  been  completely  successful  in 
identifying  and  costing  these  aspects  of  the  system.  It  is 
felt  that  the  lack  of  significance  between  the  two  categor- 
ies can  be  attributed  to  allowing  the  SAR  development 
estimate  to  be  reviewed  and  updated  when  required  once  a 
production  contact  has  been  let. 

In  the  event  that  differences  within  groups  were  obscured 
by  combination,  the  development  and  procurement  classes  were 
again  divided,  in  the  development  cast  by  the  age  of  the 
program,  and  in  the  case  of  procurement  by  whether  the  system 
had  reached  IOC.  These  two  groups  were  then  compared  and 
the  results  presented  in  Table  3-5 (b). 

TABLE  3 - 5(B) 

COMPARISON  OF  PROGRAMS  BASED  ON 
LIFE-CYCLE  DIFFERENCES 

PROGRAMS  AT/PAST  IOC  SIGNIFICANCE  RECENTLY  INITIATED  PROGRAMS 


Cost  Growth ( %) 

1.22 

NS 

Cost  Growth  (%) 

1. 

16 

Performance 

0.96 

0.025 

Performance 

0. 

99 

Schedule  Slip 
(%) 

1.29 

NS 

Schedule  Slip 
(%) 

1. 

35 

Schedule  Slip 
(Mos.  ) 

16.4 

NS 

Schedule  Slip 
(Mos . ) 

24 

.1 

Cost  Growth ($) 

523. 8M 

0.050 

Cost  Growth ($) 

266 

. 5M 

The  results  were  generally  supportive  of  those  shown  in 
Table  3 - 5(a).  Once  again  the  difference  in  cost  growth 
is  significant;  however,  the  difference  in  the  means  of 
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achieved  performance  was  unexpected.  One  possible  explana- 
tion is  that  the  realities  of  a fully  operational  deployment 
might  test  system  capabilities  better  than  a formal  evalua- 
tion period,  principally  in  the  important  areas  of  relia- 
bility and  maintainability.  On  balance,  the  findings  in 
Tables  3-5  (a}  and  3-5  (b)  supported  the  contention  that 
the  SAR  in  its  present  form  does  overcome  sufficiently  the 
difficulties  in  comparing  programs  at  various  stages  of 
completeness. 


C.  CORRELATION 

The  results  of  the  nonparametric  correlation  are  presented 
in  Tables  3-6  through  3 - 10(c).  Overall,  the  results  are 
not  impressive.  To  be  sure,  there  were  individual  cases  in 
which  the  controlled  characteristic  produced  effective 
results.  For  example,  missile  programs  displayed  an  across- 
the-board  effectiveness  in  predicting  success.  (See  Table 
3-7  (b) ) . Why  this  is  true  of  missile  systems  and  not  ships 
is  not  readily  apparent.  For  whatever  reason,  there  is  a 
greater  homogeneity  among  missile  programs  than  among  other 
groupings.  Perhaps  then,  one  could  suggest  that  greater 
faith  could  be  placed  in  the  initial  estimates  of  missile 


program  success  to  the  extent  that  they  are  valid  indicators 
of  future  results.  Characteristic,  however,  of  the  analysis 
was  the  finding  of  less  than  50%  of  the  correlations  as 


nebulous.  Additionally,  the  whole  purpose  of  ordering 
systems  by  some  specialized  criterion  variable  to  isolate 
the  effects  of  program  determinants  is  called  into  question 
when  one  recognizes  that  the  category  of  "all  programs" 
exhibited  a greater  number  of  significant  correlations  than 
a majority  of  the  other  instances  when  a program  character- 
istic, like  service  sponsor,  was  controlled.  Actually,  the 
taxonomic  analysis  was  weakened  further  by  disparities  in 
sample  size  among  some  of  the  classes  of  programs.  The 
greatest  difference  was  noted  between  the  sole-source 
contracted  programs  (40)  and  multiple-source  contracted 
programs  (7) . It  is  not  unexpected  that  those  categories 
had  by  comparison  more  significant  results  than  their  smaller 
counterparts,  since  greater  sample  size  lowers  the  level  at 
which  a relationship  between  variables  can  be  deemed  signifi- 
cant. 
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TABLE  3 


6* 


Program 

Unit 

Cost 

Program 

Cost 

Program 

Development 

Investment 

Program 

Length 


ALL  PROGRAMS 


Performance 

Factor 

Cost 

Growth 

Factor 

(%) 

Cost  Schedule 

Growth  Slip 

Factor  Factor 

($)  (%) 

Schedule 
Slip 
Factor 
(Mos . ) 

0.1193 

-0.2611 

(0.038) 

0.1430  -0.1668 

-0.0682 

0.1321 

-0.3488 

(0.008) 

0.4549  -0.5408 

(0.001)  (0.001) 

-0.3892 

(0.003) 

0.0545 

-0.1415 

-0.0964  -0.0746 

-0.0608 

0.2822 

(0.027) 

-0.2763 

(0.032) 

0.0999  -0.3898 
(0.004) 

-0.0664 

* Entries  in  parentheses  indicate 
level  to  which  Spearman  correla- 
tion is  significant.  All  others 
are  not  significant. 


Sample  Size  48 


Program 


Outcomes  (Mean) 

Performance 

0.992 

Cost  Growth  (%) 

1.210 

Schedule  Slip  (%) 

1.326 
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Performance  1 

lost 

Cost 

Schedule 

Schedule 

.ichieveuer.t 

rowth  ( ,) 

Growth  (I>) 

Slip  (%) 

Slip  (l'os.) 

Unit  Cost 

N / A 

-0 . 37b6 

-U . bo.// 

0.0b  78 

U . 54  39 

(0.05b ) 

(0.065) 

'.’oral  Program 

N/A 

-U.7486 

-0.4333 

-0  ,o27 

-0.494 

'’oat 

(0.011) 

(0.035) 

(0.088) 

Deve  iop«ient 

N/A 

0 .0891) 

-0.2447 

0.4120 

0.5297 

Investment 

(0.071) 

1 

Pro<;r<uu  Length 

N/A 

-0.3109 

-0 . 4ob3 

-0.1362 

0.3b 13 

Mean 

Outcomes 

bample  Size 

Performance 

Cost  Growth 

bcnedule  blip 

9 

L.0U0 

1.220 

1.306 

T .1  1l.c.  3 - 7(13) 


i > t . jo  X i-ii'.  >.  j / ij  CL  i >o 


1 

Performance  Cost 

Cost 

Schedule 

Schedule 

i 

Achievement  Grov.th  [% 

G i owt  h ( $ ) 

blip  ( .) 

Slip  (Mos.) 

Unit  Cost 

-0 . ‘ b 12  -0.3  >05 

0.2953 

-0.4653 

-0.3940 

j 

(0.055) 

(0.022) 

(0.048) 

Total  Program 

-L‘,01.;3  -0.4677 

0.6017 

-O.ba-19 

-U.5670 

Cost 

(0.025) 

(0.004) 

(U.Oul) 

(0.006  ) 

• . Oevelopment 

0.0432  -0.5716 

-0.18b! 

-u.3478 

-0.4118 

Investment 

(C.007) 

(0.072) 

(0.040) 

Pro.  ram  Length 

0.4-  71  -0.2992 

0.4363 

-0.7703 

-0. 4780 

. .025) 

(0.040 ) 

C.0L1) 

(0.022) 

eun  Outcomes 

..ample  l>izo 

Pe  r ! ormance 

Cost  Growth 

ich.odule  b-.n 

19 

487 

1 .22o 

1.509 

No  correlations  with  achieves  performance  were  possible  because  each 
each  of  the  programs  in  the  sample  was  assigned  a factor  of  1.0  . 
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TABLE  3 - 7(C) 


AIPCRA.-T  SYSTEMS 


Performance 
Achieveme  nt 

Cost 

Growth  (*») 

Cost 

Growth  ($) 

Schedule 
Slip  (',) 

Schedule 
Slip  (llos.) 

Unit  Cost 

-0.922 

0.1694 

0.3978 

(0.078) 

0.1046 

0.0371 

Total  Program 
Cost 

0.3709 

-0.1606 

0.5341 

(0.025) 

-0.1246 

-0.0380 

Development 

Investment 

0.1554 

0.2381 

0.0551 

0.5062 

(0.032) 

0.4285 

(0.063) 

Program  length 

0.2001 

-0.2C90 

-0.3055 

-0.1357 

-0.0112 

Mean  Outcomes 

Sample  Size 
14 

Performance 

0.991 

Cost  Growth 
1.149 

Schedule  Slip 
1.145 

TABLE  3 - 

7(D) 

MISCELLANEOUS 

SYSTEMS 

Performance 

Achievement 

Cost 

Growth  (%) 

Cost 

Growth  ( 3) 

Schedule 
Slip  (%) 

Schedule 
Slip  (”os.) 

Unit  Cost 

0.0 

u .2000 

0.5429 

-0.5000 

-0.1000 

Total  Program 
Cost 

-0 .1160 

-0.0286 

1.0000 

(0.001) 

-0.3000 

(0.052) 

-0.6000 

Development 

Investment 

0.2899 

-0.0357 

0.2571 

-0.3000 

0.1000 

Program  Length 

-0.2500 

0.2029 

0.2899 

-0.6667 

-0.3591 

Mean  Outcomes 


Sample  Size 
6 


Performance 

0.998 


Cost  Growth 
1.170 


Schedule  Slip 
1.170 


TABLE  3-3  (Al 


SOLE-SOURCE  CONTRACTED  PROGRAMS 


Performance 

Achievement 

Cost 

Growth  {“!.) 

Cost 

Growth  ( S ) 

Schedule 
Slip  (J.) 

Schedule 
Slip  (Mos.) 

Unit  Cost 

0.2001 

-0.3363 

(0.013) 

0.1433 

-0.1103 

0.0243 

Total  Program 
Cost 

0.2042 

-0.3304 

(0.020) 

0.5929 

(0.001) 

-0.4943 

(0.001) 

-0.3401 

(0.017) 

Development 

Investment 

0.0392 

-0.1106 

-0.0598 

-0.0448 

-0.0539 

Program  Lencrth 

0.2388 

(0.037) 

-0.1430 

0.2477 

(0.067) 

-0.2710 

(0.050) 

0.0464 

Mean  Outcomes 

Sample  Size 
40 

Perm 

0 

arm  nee 
.986 

Cost  Growth 
1.207 

Schedule  Slip 
1.303 

TABLE  3 - 8(B) 

MULTIPLE 

-SOURCE  CONTRACTED  PROGRAMS 

Performance 

Achievement 

Cost 

Growth  (%) 

Cost 

Growth  (3) 

Schedule 
Slip  (««) 

Schedule 
Slip  (!'os.) 

Unit  Cost 

-0.2041 

-0.6429 

(0.060) 

-0.2357 

-0.5357 

-0.3571 

Total  Program 
Cost 

-0.6124 

(0.072) 

-0.4643 

0.0357 

-0.6429 

(0.060) 

-0.5714 

(0.090) 

Development 

Investment 

0.6124 

(0.072) 

-0.2143 

0.3571 

-0.1786 

-0.0714 

Program  Length 

0.0 

-0.4140 

-0.1261 

-0.6847 

(0.045) 

-0.5225 

Sample  Size 
7 


Performance 

1.023 


Cost  Growth 
1.140 


Schedule  Slip 
1.443 


PROG RAI!S  IN  PROCUREMENT 


Performance 

Cost 

Cost 

Schedule 

( Schedule 

Achievement 

Growth  (c',) 

Growth  ($) 

Slip  (',) 

Slip  (Mos.) 

Unit  Cost 

0.0901 

-0.4680 

-0.2348 

0.0462 

0.1611 

(0.008) 

(0.079) 

^otdl  Program 

0.0619 

-0.3233 

0.4138 

-0.6198 

-0.592a 

Cost 

(0.054) 

(0.017) 

(0.001) 

(0.001) 

Development 

-0.1763 

-0.2361 

-0.1229 

-0.4241 

-0.0675 

Investment 

(0.014) 

Program  Length 

0.3805 

-0.2419 

-0.0539 

-0.3564 

-0.1602 

(0.028) 

(0.037) 

Sample  Size 

Performance 

Cost  Growth 

Schedule  Slip 

27 

0. 

976 

1.225 

1.333 

'.'ABLE  3 

- 9(B) 

PROGRAMS  IN  DEVELOPMENT 

Performance 

Achievement 

Cost 
Growth  1 

Cost 

!%)  Growth  (S>) 

Schedule 
Slip  (%) 

Schedule 
Slip  (Mos.) 

Unit  Cost 

0.3953 

(0.058) 

-0.1508 

0.3225 

0.0270 

0.0418 

Total  Program 
Cost 

0.3293 

(0.098) 

-0.4071 

(0.052) 

0.4414 

(0.033) 

-0.4880 

(0.023) 

-0.0984 

Development 

Investment 

0.5151 

(0.017) 

-0 . 3656 
(0.074) 

0.0319 

-011501 

-0.048b 

Program  *-ength 

0.3304 

(0.098) 

-0.3544 

(0.081) 

0.3790 

(0.067) 

-0.2331 

0.1953 

Sample  Size 
17 

Performance 

0.9*5 

Mean  Outcomes 
Cost  Growth 
1.178 

Schedule  Slip 
1.293 

TABLE  3 - 10(A) 


NAVY  PROGRAMS 


Performance 

Achievement 

Cost 

Growth 

Cost 

(°i)  Growth  ($) 

Schedule 
Slip  (7.) 

Schedule 
Slip  (Mos.) 

Unit  Cost 

0.3760 

(0.032) 

-0.5484 

(0.003) 

-0.2130 

-0.1329 

-0.0185 

Total  Program 
Cost 

0.2866 

(0.082) 

-0.6306 

(0.001) 

0.1304 

-0.6177 

(0.001) 

-0.5712 

(0.001) 

Development 

Investment 

0.0557 

0.0187 

-0.2331 

-0.1079 

-0.0922 

Program  Length 

0.4317 

(0.016) 

-0.5273 

(0.004) 

-0.1628 

-0.4471 

(0.013) 

-0.2337 

Mean  Outcomes 

Sample  Size 

25 

Performance 

0.969 

Cost  Growt 
1.272 

h 1 

Schedule  Slip 
1.404 

TABLE  3 - 

10(3) 

nJJMY  PROGRAMS 

Performance 

Achievement 

Cost 

Growth 

Cost 

{%)  Growth  (3) 

Schedule 
Slip  (“ ) 

Schedule 
Slip  (Mos. ) 

Unit  Cost 

-0.0305 

-0.2992 

0.0374 

-0.1923 

0.0  |j 

Total  Procram 
Cost 

»u .1997 

-0.3036 

0.5604 

(0.019) 

-0.5110 

(0.037) 

-0.0718 

1 

Development 

Investment 

0.0113 

-0.3297 

-0.3128 

C .0826 

0.2138 

Proaram  Length 

0.2744 

-0 . 150b 

-0.1084 

-0.6177 

(0.012) 

0.0501 

Mean  Outcomes 

Sample  Size  Performance  Cost  "rowth  Schedule  Slip 

14  1.025  1.149  1.292 


TABLE  3 - 10(C) 


Unit  Cost 


Total  Program 
Cost 


Level opment 
Investment 


Program  Length 


Sample  Size 
9 


USAF  PROGRAMS 


Performance 
Achieveme nt 

Cost 

Growth  (?„) 

Cost 

Growth  ($) 

Schedule 
Slip  (%) 

Schedule 
Slip  (Mos.) 

0.1219 

0.1167 

0.4167 

-0.0848 

0.0509 

0.3133 

0.0 

0.7333 

(0.012) 

-0.2(334 

-0.0170 

-0.6659 

(0.027) 

0.1303 

0.2678 

0.1532 

0.2383 

0.2283 

-0.0952 

0.5283 

-0.0479 

0.4072 

(0.091) 

Mean  Outcomes 

Performance  Cost  Growth  Schedule  Slip 

1.006  1.141  1.160 
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Table  3-6  has  displayed  the  correlation  results  for  all 
forty-eight  programs  without  any  differentiation.  The 
strength  of  program  cost  in  determining  schedule  success  is 
apparent  by  the  significance  levels  of  the  correlations 
(0.001  and  0.003).  Table  3-7  reveals  the  superiority  of 
the  missile  system  correlation  over  the  other  system  types. 
Eleven  out  of  eighteen  valid  correlations  were  significant 
in  the  case  of  missile  programs,  while  ship  systems  produced 
but  five.  Oddly  enough,  aircraft  systems  with  a larger  sample 
size  than  ship  systems  had  only  four  significant  correlations. 
Apparently,  aircraft  systems  were  responding  to  other  deter - 
inants,  most  probably  managerial  in  origin. 

Despite  the  great  difference  in  sample  size  note  pre- 
viously, multi-source  programs  and  sole-source  programs  pro- 
duced very  nearly  the  same  number  of  significant  correlations. 
(See  Table  3 - 8) . In  the  case  of  the  second  source  programs, 
the  lack  of  impact  on  the  part  of  program  cost  on  absolute 
cost  growth  rioted  earlier  was  suprising,  thereby  suggesting 
that  cost  benefits  do  occur  as  the  be  suit  of  competition.  In 
common  with  multi-source  contracted  programs,  developing 
programs  reflected  a decreasing  level  of  performance  with 
increasing  technical  risk.  This  expected  result  was 
curiously  not  demonstrated  elsewhere.  The  differences 
between  service  programs  discussed  earlier  are  exhibited  in 
Table  3 - 1C.  Not  suprisingly,  with  a larger  sample  size, 

Navy  programs  produced  more  significant  correlations  than 


either  of  the  other  services'  programs  combined. 

Table  3-11  represents  a summary  of  the  statistically 
significant  ( 0.100)  tests  presented  in  Tables  3-7  through 

3-10,  showing  the  degree  to  which  variable  relationships 
were  common  regardless  of  the  controlled  program  character- 
istic. Similarily,  Table  3-12  displays  a summary  of  the 
correlation  (not  shown)  of  dependent  variables  with  them- 
selves and  the  degree  of  commonality  encountered. 
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TABLE  3-11 


SUMMARY  OF  SIGNIFICANT  CORRELATIONS  BETWEEN 
DEPENDENT/INDEPENDENT  VARIABLES 


Performance 


Positive 


Negative 


Unit  Cost  1 
Program  Cost  9 
Development  Investment  0 
Program  Length  4 


2 

0 

0 

0 


Cost  Growth  (%) 


Positive 


Negative 


Unit  Cost  0 
Program  Cost  0 
Development  Investment  0 
Program  Length  0 


6 

7 

2 

3 


Schedule  Slip  (%) 


Positive 


Negative 


Unit  Cost  2 
Program  Cost  3 
Development  Investment  2 
Program  Length  6 


0 

1 

0 

0 


Schedule  Slip  (Mos.) 


Positive 


Negative 


Unit  Cost  0 
Program  Cost  0 
Development  Investment  1 
Program  Length  0 


1 

10 

1 

7 


Cost  Growth  ($) 


Positive 


Negative 


Unit  Cost  1 
Program  Cost  0 
Development  Investment  2 
Program  Length  0 


1 

7 

1 

1 
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TABLE  3-12 


SUMMARY  OF  SIGNIFICANT  CORRELATIONS  BETWEEN 
DEPENDENT  VARIABLES 


Performance 


Positive 


Negative 


Cost  Growth  (%)  0 
Cost  Growth  ($)  0 
Schedule  Slip  (%)  0 
Schedule  Slip  (Mos.)  0 


6 

0 

0 

0 


Cost  Growth  (%) 


Positive 


Negative 


Cost  Growth  (?)  10 
Schedule  Slip  (%)  9 
Schedule  Slip  (Mos.)  4 


0 

0 

0 


Cost  Growth  (?) 


Positive 


Negative 


Schedule  Slip  (%)  1 

Schedule  Slip  (Mos.)  1 


2 

2 


Schedule  Slip  (%)  . 


Positive 


Negative 


Schedule  Slip  (Mos.)  12 


0 


The  results  of  the  correlation  analysis  as  summarized  in 
the  above  tables  illustrate  some  interesting  points.  The 
first  of  these  is  the  weakness  of  the  independent  variable, 
development  investment.  Construed  as  a measure  of  tech- 
nological risk,  the  variable  failed  to  correlate  with  any  of 
the  outcomes  except  in  a few  cases.  One  of  two  conclusions 
can  be  reached,  either  the  program  charactersitic  of  risk 
is  valid  and  the  variable  as  defined  did  not  capture  it,  or 
risk  has  little  or  no  effect  on  program  outcome.  It  is 
suspected  that  the  former  is  true  based  upon  the  results 
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reported  by  Peck  and  Scherer. 17  Also,  on  reflection,  it  is 
felt  that  the  data  as  presented  in  the  SAR's  could  not  be 
manipulated  otherwise  to  measure  risk. 

Much  the  same  criticism  can  be  directed  toward  the  use 
or  program  unit  cost.  With  the  exception  of  relative  cost 
growth,  unit  cost  did  little  to  explain  program  outcomes. 

The  fact  that  unit  cost  did  correlate  signific  ntly  with 
relative  cost  growth  in  six  out  of  a possible  twelve  cases 
was  unexpected.  The  variable  was  intended  initially  to 
support  development  investment  as  a measure  of  the  complex- 
ity of  the  program.  Intuitively  one  would  expect  that  the 
relationship  between  unit  cost  and  cost  growth  (%)  to  be  a 
positive  one,  not  negative.  It  is  suspected  that  unit  cost 
may  be  reflective  of  an  unknown  project  feature. 

From  Table  3 - 12,  a summary  of  the  correlations  between 
dependent  variables  reveals  that  a definite  relationship 
between  cost  and  schedule  growth  exists,  perhaps  more  aptly 
put  as  "time  is  money".  As  in  the  case  of  the  analysis 
involving  both  dependent  and  independent  variables,  perfor- 
mance achievement  remained  a largely  unexplained  outcome, 
although  in  6 of  11  trials,  a significant  negative  correla- 
tion between  relative  cost  growth  and  performance  was  report- 
ed. It  may  be  hasty  to  say  that  a desire  to  reach  or  surpass 
initial  performance  levels  causes  cost  overruns  since  a 

17  Peck,  M.  J.  and  Scherer,  F.  M.,  The  Weapons  Acquisi- 
tion Process:  An  Economic  Analysis,  Chapter  16,  Research 
Division,  Graduate  School  of  Business  Administration,  Harvard 
University,  1962 
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corresponding  relationship  between  performance  and  cost 
growth  in  dollars  was  not  evident.  The  overwhelming 
number  of  significant  correlations  between  relative  and 
absolute  cost  overruns,  and  schedule  slippages  must  be 
discounted  on  definitional  grounds. 

D.  CONCLUSIONS 

From  the  results  of  the  hypothesis  testing,  three 
major  conclusions  have  been  derived. 

1 . Contrary  to  previous  belief,  large,  expensive 
programs  exhibited  a greater  controllability  in  terms 
of  schedule  than  did  projects  of  smaller  size. 

One  possible  explanation  is  that  large,  expensive  pro- 
grams attract  a great  amount  of  attention  from  every  quarter, 
the  sponsoring  service  itself,  OSD,  Congress  and  the  public. 
Conceivably,  greater  pressure  are  brought  to  bear  to  keep 
large  programs  within  the  limits  of  an  original  assessment 
as  suggested -by  the  Blue  Ribbon  Defense  Panel,  and  thereby 
avoid  the  embarassment  of  adverse  program  outcomes.  A 
second  reason  (and  one  not  necessarily  exclusive  of  the 
first)  might  be  that  large  programs,  because  of  their  size 
and  sensitivity,  are  subject  to  greater  scrutiny  before  the 
projects  get  underway,  and  thus  the  original  estimates  are 
more  realistic.  Also,  once  underway,  program  offices  may 
find  themselves  more  adequately  staffed  both  in  numbers  of 
people  and  their  expertise  to  ensure  success. 
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Although  no  judgements  were  made  of  the  manner  in  which 
development  funds  were  spent,  there  are  indications  that 
the  management  of  the  development/system  definition  phase 
of  the  life-cycle  should  be  redirected.  The  staff  report  on 
the  acquisition  process  to  the  Blue  Ribbon  Defense  Panel  in 
1970  reported  that  the  services  had  failed  to  receive 
sufficient  funding  to  carry  out  necessary  technical  work 
during  the  early  periods  of  their  programs.  What  would  often 
happen  then  was  that  critical  engineering  studies  would  be 
conducted  during  the  production  phase  when  money  was  more 
abundant.  It  is  plausible  that  an  adverse  effect  on  system 
capabilities  and  performance  would  be  suffered. 

Related  to  this  last  point  is  the  complete  lack  of  effect 
that  development  investment  displayed  when  defined  as  a 
ratio  of  development  funding  to  program  cost.  It  may  be 
entirely  possible  that  the  definition  of  development  invest- 
ment is  inadequate  to  the  task  of  measuring  technical 
uncertainty.  Despite  the  fact  that  a majority  of  programs 
(40  out  of  48)  met  or  surpassed  the  initial  performance  goals 
established  for  them,  and  that  the  "science"  of  predicting 
what  is  technically  feasible  may  be  more  efficient  than  the 
"art"  of  estimating  cost  or  schedule  outcomes,  one  should 
be  reluctant  to  surmise  that  developmental  effort  has  less 


effect  on  project  success  than  other  program  variables. 
Although  certainly  less  program  resources  are  committed 
to  advancing  the  state-of-the-art  than  manufacturing  final 
hardware,  judging  from  figures  for  the  cost  growth  of 
strictly  development  budgets  as  reported  by  Peck  and  Scherer, 
cost  variance  has  exceeded  three  or  four  times  the  original 
estimates  in  the  past.  It  is  conceivable  that  these  develop- 
mental excesses  could  have  a snowballing  effect  of  the 
remainder  of  the  program,  expecially  on  schedule  milestones 
if  stirct  measures  against  concurrent  development  and  pro- 
duction are  taken. 

3 . Program  length  is  related  inversely  to  the  achieve- 
ment goals. 

It  appears  from  an  examination  of  all  the  characteristics 
that  performance  estimates  were  either  met  or  exceeded,  and 
so  it  is  surprising  to  find  indications  that  program  length 
is  related  in  a positive  way  to  achievement  of  desired 
performance.  Peck  and  Scherer  came  across  a possible  explan- 
ation in  their  study. 

Each  engineer  wants  to  perfect  things  on  which  he 
works.  Every  development  nevertheless  involves 
compromises  between  quality  and  time.  If  schedules 
are  set  too  generously,  there  is  a tendency  for 
persons  working  on  the  progect  to  use  up  the  sche- 
duled time  and  still  be  no  nearer  completion  on 
their  hardware  goals  than  under  a shorter  schedule. 
Therefore,  in  order  to  provide  a good  target  a 
schedule  should  be  short  enough  to  make  people  think 
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and  worry  about  getting  the  hardware  out. 18 
Unfortunately,  program  length  contributed  very  little  else  to 
an  understanding  of  program  success. 


18  Peck,  M.  J.  and  Scherer,  F.  M.,  The  Weapons  Acquisition 
Process:  An  Economic  Process,  p.  413,  Research  Division, 
Graduate  School  of  Business  Administration,  Harvard  University, 
1962 


IV.  COMMENTS  AND  RECOMMENDATIONS 


Fundamental  to  this  thesis  was  the  point  of  view  from 
which  the  analysis  was  conducted.  Each  of  the  forty-eight 
programs  were  considered  as  "black  boxes”  into  which  certain 
inputs  were  made  (cost,  time,  effort,  etc.)  and  from  which 
certain  measurable  outputs  were  collected.  Meadows  and 
Marquis  criticized  such  "post  mortem"  techniques  in  their 
own  analysis  of  the  implications  of  forecasting  errors  for 
ignoring  the  interaction  between  forecasts  and  outcomes  in 
the  process  of  development.-^  Additionally,  Determinants 
of  Project  Success  by  Murphy,  et  al,  stated,  "with  rare 
exception,  the  determinants  of  project  success  were  manage- 
ment factors,  things  which  management  had  the  potential 
ability  to  inf luence . . . " , 20  a conclusion  which  necessarily 
required  a point  of  view  from  within  the  box. 

The  analysis  undertaken  here  does  not  mean  to  suggest 
that  management  techniques  are  without  effect  in  determining 
program  outcomes.  Instead,  the  study  asked  how  program 
initial  conditions  related  to  program  outcomes,  or  more 
precisely,  to  what  extent  could  the  initial  conditions  be 

1 Q 

Meadows,  D.  L.  and  Marquis,  D.  G.,  Characteristics 
and  Implications  of  Forecasting  Errors  in  the  Selection  of~ 
R&D  Projects,  paper  delivered  by  D.  L.  Meadows  at  The 
Second  Annual  Technology  and  Management  Conference,  Washing- 
ton, D.C.,  21  March  1968 

20  Murphy,  D.  C.,  Baker,  B.  N.,  and  Fisher,  D.,  Deter- 
minants of  Project  Success,  p.  5,  Management  Institute, 
School  of  Management,  Boston  College,  1974 
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relied  upon  to  "explain"  program  success.  'Explain'  is 
used  here  in  the  sense  of  signalling  the  degree  of  success 
that  a program  most  likely  would  enjoy.  And  because  of 
that.  Meadows  and  Marquis  are  correct,  the  transfer  function 
of  the  black  box  is  not  really  of  primary  concern.  Actually, 
the  results  of  this  study  do  uphold  Murphy  in  that  the 
weakness  of  the  correlation  implies  that  other  determinants 
are  at  work. 

Before  dismissing  the  methodology  as  indecisive  at  best, 
perhaps  a few  suggestions  as  to  how  to  improve  it  would  be 
in  order.  During  the  accumulation  of  data  for  the  thesis, 
a concerted  effort  was  made  to  ascertain  a measure  of  the 
relative  importance  of  each  project  without  success.  The 
Navy  practice  of  assigning  importance  classification  based 
on  the  significance  of  the  preceived  threat  to  program  once 
the  Specific  Operational  Requirement  has  been  drafted  is  no 

i 

longer  in  effect,  and  no  joint  ranking  system  appears  to 
exist.  The  addition  of  this  extra  variable  as  a measure  of 
how  the  program  is  perceived  outside  the  project  office 
would  strengthen  greatly  the  correlation  prodedure  and  no 
doubt  serve  as  the  basis  for  an  interesting  hypothesis  as 
well . 

It  is  felt  that  the  data  from  the  SAR  should  be  supple- 
mented by  information  gained  from  other  sources,  such  as 
interviews  or  questionaires . For  instance,  the  dilemma  as 
to  what  effect  technological  advance  has  on  project  success 
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could  be  settled  by  correlating  the  technical  assessment  of 
program  office  engineering  personnel  with  the  SAR  measure- 
ment. Additionally,  some  understanding  of  how  initial 
forecasts  drive  the  program,  that  is  how  the  attitude  of 
managers  toward  these  development  estimates  is  reflected  in 
the  outcome,  might  prove  an  interesting  variable  in  itself. 
Obviously  though,  the  distance  between  the  analyst  and  the 
programs  would  have  to  be  shortened . 
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APPENDIX  B 

COMPUTATION  OF  IMPLICIT  PRICE  DEFLATORS 

The  Direct  Budget  Plan  (TOA)  History  (following  page) 
was  the  source  of  the  implicit  price  deflators  discussed 
earlier.  The  following  example  will  demonstrate  how  these 
conversion  factors  were  derived. 

A - 7E  Program 

From  the  SAR  in  Appendix  A,  the  program  base 
year  is  established  as  1967.  Then,  from  the  TOA  history, 
the  ratio  of  FY  1967  obligations  to  an  equivalent  amount  of 
1977  dollars  is  computed.  This  is: 

$21 , 190M/$42 , 904M  = 0.493 

Similarily,  a ratio  of  FY  1974  obligations  to  1977  dollars 
is  also  calculated.  This  is: 

$26,88  8M/$  3 2 , 9 2 6M  = 0,817 

The  second  ratio  is  then  divided  by  the  first  and  the  result 
is  a price  deflator  for  converting  1967  dollar  amounts  to  an 
equivalency  in  1974  dollars. 

0.817/0.493  = 1.66 


79 


Department  of  the  Navy 

DIRECT  BUDGET  PI, AN  (TOA)  HISTORY 

fS  Billions) 


Note:  Totals  may  not  add  due  to  roundiny 


APPENDIX  C 

COMPUTATION  OF  OUTCOME  VARIABLES 

Listed  in  the  following  pages  are  the  forty-eight  programs 
which  constituted  the  sample  of  acquisition  programs  studied 
and  their  associated  outcomes.  Before  proceeding  further, 
it  was  felt  that  some  examples  of  hew  each  of  the  outcome 
factors  was  obtained  would  be  beneficial.  The  A-7E  SAP  in 
Appendix  A will  serve  as  a data  source, 

1.  Cost  Growth  (%) 

As  stated  previously  in  Chapter  II,  the  measure  of 
relative  cost  growth  was  obtained  directly  from  "SAP  Program 
Acquisition  Cost  Summary".  That  document  reported  the  A-7E 
project  relative  cost  growth  as  + 26%.  Thus,  the  program  has 
a cost  growth  fa  ctor  of  1.26. 

2.  Performance 

In  Chapter  II,  the  achieved  performance  factor  is 
stated  as  being  derived  as  a weighted  average  of  key  per- 
formance criteria.  In  the  case  of  the  A-7E  program,  these 
were : 

1.  maximum  speed  at  sea  level; 

2.  radius/range  during  primary  attack  mission; 

3.  ceiling/altitude; 

4.  reliability; 

5.  maintainability  - 

a.  Scheduled  Depot  Level  Maintenance  (SDLM) 

b.  Maintenance  Man-Hour  per  Flight  Hour. 
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A comparison  of  the  current  estimate/demonstrated  performance 
and  the  development  estimate  for  each  of  the  criteria  above 
yields  the  following  results. 

1.  505  KTS/555  KTS  = 0.91 

2.  537  NM/ 475  NM  = 1.13 

3.  35,500  ft/35,000  ft  = 1.00 

4.  0.78/0.84  - 0.93 

5a.  24  MOS/30  MOS  = 0.80 
b.  5.4  HR/9.6  HR  = 0.56 

(Weighted  equally,  maintainability 
is  0.68) 

Summing  all  of  the  scores  and  dividing  by  the  number  of 
criteria,  one  achieves  a performance  factor  of  0.93  for  the 
program. 

The  above  procedure  is  illustrative  of  the  general 
approach  used.  There  were  two  modifications  to  this  method, 
one  in  the  case  of  ships,  the  other  in  the  case  of  certain 
one-shot  devices.  Unlike  aircraft  where  speed,  maneuver- 
ability, range,  reliability  and  maintainability  taken  toge- 
ther represent  a meaningful  measure  of  ystem  performance, 
a comparison  of  SAR  criteria  for  ships  (e.g.,  speed,  manning 
level,  displacement,  etc.)  does  not.  Invariably  these 
criteria  are  met.  The  existence  of  multiple  mission  areas  for 
a single  ship  type  further  complicates  the  figure  of  merit 
calculations.  This  is  a good  example  of  where  system 
ewectiveness  breaks  down  in  a computational  sense.  For  these 
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reasons,  ship  systems  were  arbitrarily  assigned  a performance 
factor  of  1.0. 

One-shot  systems  (less  strategic  systems)  were  evaluated 
based  on  a comparison  of  their  single  shot  probability  of 
kill  given  in  the  SAR.  This  assumed  the  systems  were  activated 
within  their  proper  search/attack  envelopes  and  that  all 
components  functioned  properly. 

3.  Schedule  Growth  (%) 

From  the  Schedule  Milestone  section  of  the  SAR,  the 
program  was  initiated  in  May  1963  and  achieved  an  IOC  on 
schedule  in  April  1970.  Thus,  there  was  no  schedule  slippage 
and  a factor  of  1.0  was  awarded. 


PROGRAM  OUTCOME  SUMMARY  BY  SERVICE 


NAVY 


Program 

Performance 

Cost  Growth (%) 

Schedule  Slip(%) 

1. 

S-3A 

1.04 

1.03 

1.00 

2. 

Trident 

1.00 

0.99 

1.14 

3. 

Poseidon  C-3 

1.00 

N/A 

1,07 

4. 

SSN  688 

1.00 

1.01 

1.15 

5. 

Trident  C-4 

1.00 

1.06 

1.13 

6. 

DD  963 

1.00 

1.05 

1.24 

7. 

CGN  38 

1.  00 

1.05 

1.24 

8. 

Harpoon 

1.00 

1.10 

1.13 

9. 

LHA 

1.00 

1.18 

1.87 

10. 

CVAN  68 

1.00 

1.16 

1.35 

11. 

CVAN  69 

1.00 

1.25 

1.24 

12. 

Condor 

0.94 

1.46 

3.29 

13. 

CH-53E 

0.97 

1.08 

1.32 

14. 

MK  48-1  . . 

0.98 

1.19 

1.14 

15. 

P-3C 

0.99 

1.17 

1.05 

16. 

Captor 

1.  00 

1.56 

1.33 

17. 

Sparrow  III 

1.00 

1.64 

2.80 

00 

H 

PHM 

1.00 

1.84 

1.41 

19. 

Phoenix 

1.00 

1.43 

1.06 

20. 

FFG  7 

1.00 

1,45 

1.13 

21. 

A-7E 

0.93 

1.26 

1.00 

22. 

E-2C 

0.89 

1.36 

1.03 
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Program 

Performance 

Cost  Growth (%) 

Schedule  Slip(%) 

23. 

Phalanx 

0.85 

1.37 

1.36 

24. 

F-15A 

0.84 

1.22 

1.16 

25. 

Sidesinder 
(AIM  9-L) 

0.79 

1.61 

2.45 

MEAN 

0.969 

1.272 

1.404 

ARMY 

Program 

Performance 

Cost  Growth (%) 

Schedule  Slip(%) 

1. 

XM  198 

1.16 

1.03 

1.30 

2. 

Stinger 

1.62 

1.22 

1.44 

3. 

Hawk 

1.04 

1.39 

1.25 

4. 

UTTAS 

1.00 

0.88 

1.02 

5. 

Roland 

1.  00 

1.00 

1.23 

6. 

XM  1 

1.00 

1.08 

1.00 

7. 

Patriot 

1.00 

1.07 

1.21 

8. 

AAH 

1.00 

1.15 

1.33 

9. 

MICV 

1.00 

1.19 

1.34 

10. 

Dragon 

1.00 

1.19 

1.50 

11. 

Copperhead 

0.93 

1.18 

1.11 

12. 

TOW 

0.86 

1,21 

1.41 

13. 

Lance 

0.86 

1.12 

1.65 

14. 

TACFIRE 

0.88 

1.38 

N/A 

MEAN 

1.025 

1.149 

1.292 
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Program 

Performance 

Cost  Growth (%) 

Schedule  Slip(%) 

1. 

F-16 

1.06 

1.00 

1.00 

2. 

B-l 

1.12 

1.13 

1.35 

3. 

A- 10 

1.03 

1.22 

1.13 

4. 

E-3A 

1.02 

1.16 

1.10 

5. 

EF-llA 

1.00 

0.94 

1.00 

6. 

Maverick 

1.  00 

1.02 

1.22 

7. 

E-4 

1.00 

1.40 

1.61 

8. 

F-l  4 

1.00 

1.25 

1.03 

9. 

Minuteman  III 

0.82 

1.15 

1.00 

MEAN 

1.  006 

rmr 

1.160 

OVERALL  MEAN 

0.992 

1.210 

1.326 
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